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THE  ANALYSIS  OF  SEQUENTIAL  EXPERIMENTS 
WITH  FEEDBACK  TO  SUPJECTS 

Persi  Diaconis  and  R.  L.  Graham 


1.  Introduction 

For  a  variety  of  testing  situations  the  following  experiment 
is  performed:  A  subject  tries  to  guess  the  outcome  of  a  sequence  of 
draws  without  replacement  from  a  finite  population.  After  each 
guess,  the  subject  is  given  feedback  information.  This  might  be  the 
name  of  the  object  just  guessed  at — complete  feedback — or  only  the 
information  that  the  guess  just  made  was  correct  or  not — partial 
feedback.  We  are  interested  in  the  subject's  optimal  strategy  and  in 

m 

methods  for  scoring  subjects  which  do  not  depend  on  the  strategy  used 
.  by  the  subject. 

The  following  example  illustrates  our  main  results. 

1.1.  Taste  testing  and  partially  randomized  clinical  trials 

Consider  Fisher's  famous  Lady  tasting  tea  (Fisher  (1949)  p*  ll) • 
Eight  cups  of  tea  were  prepared — four  of  one  type  and  four  of  a 
second  type.  The  cups  of  tea  were  presented  to  the  lady  in  a  random 
order,  and  she  was  to  guess  the  type  for  each  cup.  With  no  ability 
and  no  feedback,  the  lady  is  expected  to  have  four  of  her  eight 
guesses  correct.  We  propose  the  following  variation:  to  help  cali¬ 
brate  her  guesses,  the  lady  is  told  after  each  guess  if  it  was 
correct  or  not.  If  the  lady  has  no  tasting  ability  but  is  trying  to 
•  maximize  the  number  of  correct  guesses,  her  optimal  strategy,  knowing 
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that  a  of  type  one  and  b  of  type  two  remained*  is  to  guess  the  type 
corresponding  to  max(a,b).  The  expected  number  of  correct  guesses 
under  the  optimal  strategy  is  373/70  ==  5.3. 

Mathematically,  this  problem  is  the  same  as  a  problem  dis¬ 
cussed  by  Blackwell  and  Hodges  (1956)  and  Efron  (1971)  in  connection 
with  clinical  trials.  In  comparing  two  treatments  on  2n  patients, 
suppose  it  is  decided  that  n  patients  are  to  get  each  treatment,  the 
allocation  being  otherwise  random.  Assume  that  the  patients  arrive 
sequentially  and  must  either  be  ruled  ineligible  or  assigned  one  of 
the  two  treatments.  A  physician  observing  the  outcome  of  each  trial 
would  know  which  treatment  was  most  probable  on  each  trial.  This 
information  could  be  used  to  bias  the  experiment  if  the  physician 
ruled  less  healthy  patients  ineligible  on  trials  when  a  favored  treat¬ 
ment  was  more  probable.  A  natural  measure  of  the  selection  bias  is 
the  number  of  correct  guesses  the  experimenter  can  make  by  guessing 
optimally.  Blackwell  and  Hodges  showed  that  with  2n  subjects  the 
optimal  guessing  strategy  leads  to 

n  +  ■|(2^n/(^')  -1)  =  n  +  */7rn  -  +  0(~)  correct  expected  guesses. 

The  same  problem  arises  in  card-guessing  experiments.  The 
usual  ESP  experiment  uses  a  25-card  deck  with  the  5  symbols  0,  +, 

///,  1  | ,  *  repeated  five  times  each.  The  deck  is  shuffled;  a 

sender  looks  at  the  cards  in  sequence  from  the  top  down,  and  a  sub¬ 
ject  guesses  at  each  card  after  the  sender  looks  at  it.  We  discuss 
three  types  of  feedback: 
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case  1 — No  feedback.  If  no  feedback  is  provided,  then  any  guessing 
strategy  has  five  correct  guesses  as  its  expected  value.  The  distri¬ 
bution  of  the  number  of  correct  guesses  depends  on  the  guessing 
strategy.  Several  writers  have  shown  that  the  variance  is  largest 
when  the  guessing  strategy  is  some  permutation  of  the  25  symbols. 

This  is  further  discussed  at  the  beginning  of  Section  3. 

Case  2 — Complete  feedback.  If  the  subject  is  shown  the  card  guessed 
each  time,  then  the  optimal  strategy  is  to  guess  the  most  probable 
remaining  type  at  each  stage.  The  expected  number  under  the  optimal 
strategy  is  8.65,  a  result  first  derived  by  Read  (1962).  In  Section 
2  we  give  closed  form  expressions  for  the  expected  number  of  correct 
guesses  for  the  optimal  and  worst  case  strategies  for  a  deck  of 
arbitrary  composition. 

Case  3 — Yes  or  no  feedback.  The  situation  becomes  complex  with  par¬ 
tial  feedback — telling  the  subject  if  each  guess  was  correct  or  not. 
No  simple  description  of  the  optimal  strategy  is  known.  An  example 
in  Section  3  shows  that  the  "greedy  algorithm"  which  guesses  the  most 
probable  symbol  at  each  stage  is  not  optimal.  The  optimal  strategy 
and  the  expected  number  of  correct  guesses  under  the  optimal  strategy 
can  be  determined  by  solving  a  recurrence  relation  numerically.  For 
a  standard  ESP  deck  the  expectation  is  6.63  correct  guesses.  In 
Theorems  5  and  6  we  show  that  the  greedy  algorithm  is  optimal 
for  partial  feedback  experiments  with  no  repeated  values  (that  is, 
for  a  deck  labeled  (1,2, . . . ,n)) .  For  an  empirical  attempt  to  solve 
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these  problems,  see  Thouless  (1977).  A  thorough  discussion  of 
statistical  problems  in  ESP  research  may  be  found  in  Burdick  and 
Kelly  (1978),  and  Diaconis  (1978). 

How  should  feedback  experiments  be  evaluated?  Consider  a 
numerical  example  made  explicit  in  Table  1.  A  deck  of  20  cards,  10 
labeled  "red"  and  10  labeled  "black,"  was  well  mixed.  A  sender 
looked  at  the  cards  in  sequence  from  the  top  down,  and  a  subject 
guessed  at  each  card  after  the  sender  looked  at  it.  After  each  trial 
the  guesser  was  told  whether  the  guess  was  correct  or  not.  There 
were  14  correct  guesses.  If  this  experiment  was  naively  evaluated  by 
neglecting  the  availability  of  feedback  information  (a  widely  used 
approach,  see  Tart  (1977),  Chapters  1,2  for  references),  each  trial 
would  be  regarded  as  an  independent  binomial  variable  with  success 
probability  1/2.  Binomial  tables  show  that  P(14  or  more  correct  out 
of  20)  =  .058.  The  choice  sequence  that  the  guesser  actually  made  is 
fairly  close  to  the  optimal  strategy.  There  were  7  times  that  the 
number  of  red  cards  remaining  was  equal  to  the  number  of  black  cards 
remaining.  At  these  trials,  red  and  black  have  the  same  probability 
of  being  correct  and  either  choice  is  optimal.  The  guesses  made 
agree  with  the  optimal  strategy  on  9  of  the  13  remaining  trials. 
Perhaps  the  14  correct  guesses  should  be  compared  with  12.30,  the 
expected  number  of  correct  guesses  under  the  optimal  strategy. 
Neglecting  the  availability  of  feedback  information  can  lead  to 
crediting  a  subject  using  an  optimal  (or  near  optimal)  strategy  with 
having  "talent."  On  the  other  hand,  demanding  that  a  subject 
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TABLE  1 


EXAMPLE  OF  SKILL  SCORING  IN  AN  EXPERIMENT  WITH 
10  RED  AND  10  BLACK  CARDS  AND  FEEDBACK 
TO  THE  GUESSING  SUBJECT 


Trial  No. 

Guess 

Feedback 

Optimal 

Pi 

Card 

1 

B 

Yes 

Tie 

1/2 

B 

2 

B 

No 

R 

9/19 

R 

3 

B 

No 

Tie 

1/2 

R 

4 

B 

Yes 

B 

9/17 

B 

5 

R 

No 

Tie 

1/2 

B 

6 

B 

Yes 

R 

7/15 

B 

7 

R 

Yes 

R 

8/14 

R 

8 

B 

Yes 

R 

6/13 

B 

9 

R 

Yes 

R 

7/12 

R 

10 

R 

Yes 

R 

6/11 

R 

11 

R 

No 

Tie 

1/2 

B 

12 

R 

Yes 

R 

5/9 

R 

13 

B 

No 

Tie 

1/2 

R 

14 

R 

Yes 

B 

3/7 

R 

15 

B 

Yes 

B 

4/6 

B 

16 

B 

Yes 

B 

3/5 

B 

17 

B 

No 

Tie 

1/2 

R 

18 

B 

Yes 

B 

2/3 

B 

19 

R 

Yes 

Tie 

1/2 

R 

20 

B 

Yes 

B 

1 

B 

14 

11.049 

Correct 

Column  1  is  trial  number.  Column  2  is  subject's  guess.  Column 
3  is  feedback  information.  Column  4  is  optimal  guess  (tie  means  either 
color  is  optimal).  Column  5  is  probability  that  subject's  guess  is 
correct,  and  Column  6  is  card  actually  present. 
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significantly  exceed  the  expected  number  under  the  optimal  strategy 
can  lead  to  failure  to  detect  a  "talented"  subject  who  doesn't  use 
the  feedback  information.  In  Section  4  we  describe  a  method  of  eval¬ 
uation  called  skill  scoring.  The  skill  score  compares  the  number  of 
correct  guesses  to  a  base  line  score  calculated  from  the  conditional 
expectation  of  the  i^  guess  given  the  feedback  information.  The 
statistic  is  particularly  simple  in  the  present  example.  If  at  the 
time  of  the  i**1  guess  there  are  r^  red  cards  and  b^  black  cards 

remaining  in  the  deck,  then  the  probability  of  the  next  card  being 
r. 

(say)  red  is  - ~i  +  i  *  T*ie  numbers  p.^ — the  probability  of  the 

iC^  guess  being  correct — are  gx._  in  the  fifth  column  of  Table  1. 

til 

If  Z^  is  one  or  zero  as  the  i  guess  is  correct  or  not,  then  the 

skill  score  statistic  S  is  defined  as  S  =  E.n.  (Z.  -p.}.  For  this 

i=l  l  *i 

example  S  =  14-11.049  =  2.95. 

In  Theorem  7  we  show  that  for  any  guessing  strategy 
S//2n 74  has  a  limiting  standard  normal  distribution.  In  the  example 
of  Table  1,  S//5  =  1.32.  Further  discussion  of  this  example  is  in 
Section  4. 

Clearly  experiments  which  combine  feedback  with  sampling  with 
replacement  are  easier  to  analyze.  Our  motivation  for  considering 
sampling  without  replacement  is  twofold.  First,  reanalysis  of  a 
previously  performed  feedback  experiment  done  without  replacement  may 
be  desirable.  Second,  experiments  are  often  designed  without 
replacement  to  insure  balance  between  treatments  for  moderate 
samples.  Efron  (1971)  gives  a  nice  discussion  of  these  issues  and 
references  to  standard  literature. 
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2.  Complete  Feedback  Experiments 


In  this  section  we  consider  experiments  with  a  deck  of  n  cards 

r 

containing  c^  cards  labeled  i,  1  <  i  £  r,  so  n  =  c^.  We  write 

c  =  (cx*c2» • *  * »cr)  f°r  the  composition  vector.  A  subject  tries  to  guess 

what  card  is  at  each  position  and  after  each  guess  is  shown  the  card  at 

this  position.  The  optimal  strategy  for  a  subject  trying  to  maximize 

the  total  number  of  correct  guesses  is  to  guess  the  most  probable  symbol 

at  each  stage.  (This  is  easily  proved  by  backward  induction.)  Let 

H  =  H(c)  be  the  number  of  correct  guesses  when  the  optimal  strategy  is 

used.  We  can  derive  the  distribution  of  H  when  r  =  2  by  using  variants 

of  an  argument  in  Blackwell  and  Hodges  (1957) .  We  give  the  limiting 

distribution  of  H  here,  the  exact  distribution  is  derived  in  the  course 
of  the  proof. 

Theorem  1. 

If  c^  and  tend  to  infinity  in  such  a  way  that 
Cl/<C1  +  C2>  ■*P»0<P<1»P^  then 

(2.1)  E(H)  =  max(c, ,c_)  +  -|(-t — - — r  -  1)  +  o(l)  . 

12  2  1  p  -  q  l 

(2.2)  P(H -maxfc^^)  =  k)  -»-Y(l-Y)^ 

for  k  =  0,1,2,...  where  y  -  .  — . 

1  +  jp  -  q| 

If  =  c^  =  k  (so  p  =  h)t  then,  as  k  tends  to  infinity, 

=  k  +  |/nk  -  \  +  0(i)  , 


(2.3) 


E(H)  =  k  +  ~  ,77'  -  1 
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(2.4  ) 


if  x  <  0 
if  x  >  0 


where  4>(x)  is  the  standard  normal  cumulative  distribution. 

Results  (2.3)  and  (2.4)  are  essentially  given  by  Blackwell 

and  Hodges  (1957).  The  results  show  that  there  is  a  big  difference 

between  balanced  decks  where  c^  =  c^  and  unbalanced  decks.  In  the 

unbalanced  situation  the  optimal  strategy  does  not  do  much  better 

than  the  strategy  which  always  guesses  the  type  corresponding  to 

raax(c^,C2).  An  intuitive  explanation  is  that  when  c^  »  C2  the 

optimal  guess  will  almost  always  guess  type  1. 

When  r  >  2,  we  have  not  actively  pursued  the  problem  of 

finding  the  distribution  of  H,  but  we  have  determined  the  mean  of  H. 

If  h(c)  *  E(H(c)),  then  elementary  considerations  show  that  when 

Z.  ,  c.  >  0,  h  satisfies  the  recursion 
i=l  1 

(2.5)  h(c)  =  Z  -----  Ci  -  h(c-  5.)  +  1g^c-)--  »  h<0>  =  0  » 

1  1  r  1  r 


where  6^  has  a  one  in  the  i  position  and  zeros  elsewhere,  and  0  is 
the  vector  of  all  zeros. 

We  will  show  that  h(c)  has  the  following  closed  form 
expression: 


Theorem  2.  The  solution  of  the  recursion  (2.5)  is 


(2.6)  h(c)=max(c)+  Z 

0^1 1  '  11 


max  (i) 

i,  +  ...  +  i 
1  r 
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WlBWStlW'i 


where 


IM»X*(t) 


0  If  there  la  <i  unique  J  aueh  that  I, ^  -  max(t) 
mnx(l)  othurwl.no 


Tito  'turn  In  (2.6)  In  over  the  nonnugotive  orthant  of  flu*  integer 
1  a  1 1 1  oo  i  u  r  d  l  mena  l  one . 

Thu  recursion  (2,5)  won  used  l>y  head  (.1962)  to  numerically 
determine  certain  values  of  li.  We  recomputed  the  following  values  of 
h (o)  confirming  Hoad's  calculations:  h (3 . 3 , 3)  <-  6.78690* , 
h(5,5, 5,5,5)  -  8.66675*.  A  direct  probabl  1  Inf  It:  Incerpretnt  ion  of 
the  right  ulilo  of  (2.6)  In  given  after  the  proof. 

Pot*  a  dunk  containing  r  dlffureut  typott  with  each  typo 
repeated  k  tlmoa,  o  =  kl,  where  1  In  a  vector  of  r  onea.  For  large 
k,  weak  convergence  teehnlquun  can  bo  unod  to  hound  the  right  aide  of 
(2.6): 


Theorem  3.  Aa  k  tonda  to  Infinity, 

h  (k  I )  -  k  4*  4  M  ,/k  ■(*  0.(/k)  , 

where  M  la  the  expected  value  of  the  maximum  of  r  .Independent, 
standard  normal  varlaf.ua.  The  notation  o  meana  the  Implied  constant 
dependa  on  r. 

The  nnmhera  M  are  tabled  In  To leh mow  (1,936)  and  Harter 
(1961).  Per  example. 


M  .566  .86)  1.029  1.163 

r 


Of  course,  Theorem  3  agrees  with  (2.3)  when  r  =  2.  When  k  =  r  =  5, 
the  approximation  given  by  Theorem  3  is  about  9.08  as  compared  with 
8,65  from  exact  evaluation. 

In  the  complete  feedback  problem  it  is  possible  for  a  subject 
to  try  to  minimize  the  expected  number  of  correct  guesses  by  guessing 
the  least  probable  symbol  on  each  trial.  We  call  this  worst  case 
guessing.  This  can  lead  to  strategies  with  a  strange  appearance. 

For  example,  with  n  cards  labeled  11, 2,..., n)  the  worst  strategy 
guesses  any  card,  (say  1)  on  the  first  trial  and  thereafter  guesses 
a  card  known  not  to  be  in  the  deck.  This  leads  to  1/n  as  the  expected 
number  of  correct  guesses.  Analysis  of  worst  case  guessing  is  vai*-'.- 
ble  in  determining  how  widely  the  distribution  of  correct  guesses  can 
vary  as  a  function  of  strategy.  The  arguments  are  similar  to  best 
case  guessing  and  will  not  be  given  in  detail.  Here  are  some  results: 

Theorem  4.  Let  d(c)  denote  the  expected  number  of  correct 
guesses  when  the  worst  case  strategy  is  used  with  complete  feedback. 


where  min'*(i)  =  min(i)  (mul(i)  -  1)  ,  mul(i)  is  the  number  of  j  such 
that  i  =  min(i) . 

As  k  tends  to  infinity, 

d(kl)  =  k  -  ^  MrA  +  o  (^c) 

where  M  was  defined  in  Theorem  3. 

11 


Some  numerical  values  for  d  are  d(3,3,3)  =  1.48690+, 
d(5,5,5,5,5)  =  2.29606+.  When  r  =  2,  min^^)  -  d  =  h  -  maxCc^c^) 
so  (2.1)  and  (2.3)  can  be  used  for  similar  computations  involving  d. 

Theorems  3  and  4  show  that  with  a  bounded  number  r  of 
distinct  types  the  deviation  of  either  best  or  worst  case  guessing 
from  guessing  with  no  feedback  is  of  order  /ic  compared  to  a  lead 
term  of  k.  This  is  crucial  to  results  in  Section  4  involving  the 
skill  scoring  statistic. 


Proofs  for  Section  2 

Proof  of  Theorem  1.  To  determine  the  distribution  of  H  we  follow 
Blackwell  and  Hodges  (1957)  in  considering  an  associated  random  walk. 
Without  loss  of  generality  suppose  c^  C2*  Following  the  notation 
of  Chapter  3  of  Feller  (1968),  consider  a  random  path  composed  of 
lines  of  slope  ±  1.  The  walk  moves  up  if  a  card  of  type  1  is  turned 
up,  and  down  if  a  card  of  type  2  turns  up.  The  walk  begins  at  (0,0) 
and  ends  at  (c^c^  ^-Cj).  The  optimal  strategy  is  to  guess 
type  1  if  the  path  is  below  the  line  y  ~  c^  -  C2,  guess  type  2  if  the 
path  is  above  this  line,  and  guess  arbitrarily  at  points  where  the 
path  touches  the  line.  This  is  because  when  the  path  touches 
y  =  ci  -  c2,  the  number  of  cards  of  type  1  remaining  equals  the  num¬ 
ber  of  cards  of  type  2  remaining.  Let  T  be  the  number  of  times  the 
random  path  touches  the  line  y  =  c^  -  C2*  It  is  not  hard  to  show  by 
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induction  that  for  any  path  the  number  of  correct  guesses  that  the 
optimal  strategy  makes  at  time  c^  +  c ^  equals  c^  +  Z  where  Z  is  a 
binomial  random  variable  with  parameters  1/2  and  T.  Thus  all  ran¬ 
domness  in  the  outcome  of  a  run  through  the  deck  using  the  optimal 
strategy  can  be  attributed  to  the  outcome  of  guesses  when  the 
remaining  numbers  of  each  type  were  the  same. 

T  takes  values  0,1,2, ... ,C2  and  a  straightforward  variant 
of  the  proof  of  Theorem  4  in  Section  7  of  Feller  (1968)  shows 


that 


(2.7)  p(T  =  t)  =  2 


t  ci-c2  +  t  fcl  +  c2-t 


cl  +  c2-t 


c2_t 


C1  +  C21 


,  t  —  0,1, ...,c 


2  * 


Notice  that  when  c^Cj,  T  cannot  take  on  the  value  0  and 


(2.7)  is  equivalent  to  equation  (2.3)  of  Blackwell  and  Hodges  (1957). 
They  argue  that  T//k  tends  in  distribution  to  the  absolute  value  of  a 
standard  normal,  and  this  implies  (2.4).  Passing  to  the  limit  in 
(2.7)  when  c^  and  C2  tend  to  infinity  with  c^/ (0^  +  02)  -►  p 
0<p<  1,  V  f  \  yields  that  T  has  a  limiting  geometric  distribution 
with  p(T  =  t)  =Y(l~Y)t  t  =  0,1,2, ...  ,y  =  |p  -  q|  .  The  limiting  distri¬ 


bution  of  H  is  obtained  from  the  limiting  distribution  of  T  by  using 


the  fact  that,  if  H  given  T  =  t  is  binomial  with  parameters  ■=•  and  t. 


then  H  unconditionally  has  the  distribution  specified  by  (2.2).  The 
equation  for  the  mean  of  H  can  be  derived  as  a  special  case  of  (2.6). 
Thus,  when  r  =  2,  max  (i^,^)^  unless  i^^-  Then  (2.6)  becomes 


(2.8) 


E(H)  =  max(c1,c2)  + 


I 
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When  =  k,  we  have 


E®  -k+5j1  fij  /  (li 


so  (2.3)  follows.  Taking  the  limit  in  (2.8)  as  — — - »■  p  yields 

cl  +  c2 

E(H)  =  max(c1,c2)  + -|  l  ^pqj  1  +  o(l) 


=  max 


=  max 


(e^)+^fe-l)  +  o(1) 

(ci>c2)  (fHt1)  +  0<1) 


Proof  of  Theorem  2.  Let  f(c)  =  h(c)  -  max(c).  The  recursion  (2.5) 
translates  into 

c ,  _  _  _  _ 

1  ■ — 1  *  (f  (c  -  6.)  +max(c  -  5  )}  +  - 
.  w.  ■  •••*»  c  i  i 

i  1  r  1 


f  (c)  =  l  — — — — —  *  (f  (c  -  6.)  +max(c  -  6.)}  +  -  - max(c) 

.  C,  +  ...+C  1  1  C,  +  .  .  .  +  C 


or 


(c,  +  ...+c  )f(c)  =  Ic.  f(c-5.) 
1  r  .x  x 

i 


+  [£  c .  max(c  -  6^)  +  max(c)-(c^  + . . . +  cr)max(c) ] 
i 


The  expression  in  square  brackets  is  easily  seen  to  equal  max  (c)  as 
defined  in  Theorem  2.  Now,  writing 


_  (c1  +  . . .  +  c  )  l  _ 

sM  -  — h - -V-  «c)  , 


the  recursion  becomes 
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(2.9) 


^  V  .  'W1 .  r  max  (c) 

g(c)  =  +  c  j  ;..t  •  '  ^7+77 7TT 

i  1  r  1  r 


It  is  clear  from  (2.9)  that  g(c)  can  be  expressed  as  a  sum  over  the 
nonnegative  orthant  0  ^  i  <  c  of  the  function 


(i,  +  ...  +i  ) ! 

1  r  max  (i) 


At  each  lattice  point  i  the  function  A ( i)  must  be  multiplied  by  the 
number  of  paths  from  c  to  i.  This  number  is 


((c^  -  ij^)  +  . . .  +  (cf  -  ir>) ! 
(Cj-ij)!  ...  (cr-ir)I 


Thus, 


((c  -  i.)  +  ...  +  (c  -  i  ))!  _ 

*®*  !  (/-i  vi - 

0?fi<i  11  r 

Transforming  g  back  to  f  and  f  back  to  h  completes  the  proof  of 
Theorem  2. 

By  considering  a  multidimensional  random  walk,  taking  a  step 

in  the  direction  of  the  i^1  coordinate  when  a  card  of  type  i  is  exposed 

•  ,  ^ 
we  can  give  a  direct  probabilistic  interpretation  to  the  max  of 

* 

Theorem  2  and  min  of  Theorem  4.  Just  as  when  r  =  2,  the  only  random¬ 
ness  in  the  number  of  correct  guesses  under  the  optimal  strategy  comes 

from  lattice  points  i  where  max  (i)  >  0.  The  number  of  correct  guesses 

*  —  _ 

from  lattice  points  where  max  (i)  =  0  being  max(c).  The  probability 

of  a  correct  guess  for  a  lattice  point  where  max  (i)  >  0  is 
*  — 

max  (i)  .  .  . 

1  t - TT“  »  an“  the  sum  in  (2.6)  is  just  a  sum  of  these 


probabilities  multiplied  by  the  probability  that  the  path  passes 
through  i. 


Proof  of  Theorem  3.  We  are  considering  a  deck  of  n  =  rk  cards 

containing  k  cards  marked  i,  1  _<  i  <  r.  For  j  =  l,2,...,n,  let  V.  be 

an  r-dimensional  random  vector  which  counts  how  many  of  each  type 

have  been  called  before  time  j.  Thus,  =  0  and  V\(i)  is  the  number 

of  cards  marked  i  which  have  appeared  before  time  j.  At  the  .j  trial 

the  optimal  strategy  is  to  choose  any  value  Z  such  that  V  (£.)  = 

min  V.(i).  The  probability  of  a  correct  guess  is  then 
i  3 


(2.10) 


k  -  min  V. (i) 
i  J 
n  -  i  +  1 


j  =  1,2 


,n  . 


To  work  with  (2.10)  we  use  weak  convergence  techniques  from  Chapter  4 

of  Billingsley  (1968).  The  first  step  is  to  transform  the  random 

vectors  V, ,...,V  into  a  random  function  which  will  be  shown  to  con- 
1  n 

verge  to  an  appropriate  Brownian  bridge.  Let 


X. 

3 


r 

k(r  -  1) 


(V. 

3 


The  components  of  X.  have  E(X.(i))  =  0,  Var(X.(i))  -  1.  Form  a 
3  3  3 

vector  valued  continuous  function  X  :  [0,1  ]  -*■  R  by  connecting  the 

n  t 

components  X^(i)  by  straight  lines  as  in  Billingsley  ((1968),  pp.  8-15). 

Thus,  X.,  =  X..  It  follows  from  Rosen's  (1967)  results  for  depen- 

n  J/n  3 

dent  vector  valued  random  variables  that  the  r-dimensional  analog  of 

—  D  _0 

Theorem  24.1  of  Billingsley  (1968)  holds.  That  is,  nXfc  -*•  Wfc  where 
is  an  r-dimensional  mean  0  Gaussian  process  with  the  following 


covariance: 


for  s  <  t 


,  E{W°(i)W?(j)}  =  { 


;  -s(l-t) 
s  -  1 


when  i  ^  j 


s (1  -  t)  when  i  =  j 


Thus,  each  component  process  W^(i)  is  a  Brownian  bridge  and,  for 


fixed  t,  cov  =  t(l-t)  $  where 


t  - 


1 

r-1 


1  *. 

r-1 


This  implies  that  l  Wt(i)=0.  Returning  to  (2.10)  and  summing  yields 


n  k  -  A 


(2.11) 


.  .  n  -  j  +  1 

3=1 


,  _  min  X.(i) 

r_  _  /k(r  -  1)  £  _i _ 


1  j-i  "‘J+1 


The  first  sum  in  (2.11)  is  easily  seen  to  equal  k  + 


■'(V) 


(the 


notation  0^.  means  that  the  implied  constant  depends  on  r).  We  will 
argue  that  we  may  take  expectations  in  (2.11)  and  pass  to  the  limit 
as  k  tends  to  infinity.  Then, 


(2.12) 


min  X.(i) 
n  <  3 


3=1 


n  -  j  +  1 


rl  E(min  W^) 

t  1't 


dt  . 


Assuming  the  validity  of  (2.12)  for  the  moment,  we  have  shown  that  the 
expected  number  of  correct  guesses  is 


k  -  M  A  +  o  (v^k) 
r  r 


where 


m  -  /£3 
r  r  Jo 


rl  E(min  W7) 


1  -  t 


dt 


We  now  show  that  >1  -  -  where  uas  defined  in  Theorem  3.  To 

prove  this  note  that  one  way  of  constructing  W^  from  r  independent 
1-dimensional  Brownian  bridges  W^,...,W^r^  is  as  follows. 
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jr  m 

Let  W  =  -  E  W(j)  and  let  W°(i)  =  / (wfl)  -Wj  for  1  <  i  <  r. 
tr.,t  t  r-lt  t  -- 

j=l 

It  is  easy  to  check  that  K  has  the  correct  covariance,  t(l-t)$.  Now, 
for  fixed  t  the  symmetry  of  mean  0  Gaussian  variables  implies  that 

E{tnin  W^*^}  =  -  E{max  W^^l;  E(min  U^(i)}  =  -  Efmax  W^(i) } 
i  i  i  i 


Moreover, 


2E(max  W^(i)}  =  E{Range{W^(i) }}  =  ^  E{Range  w£*^} 


=  2 / E{max  wf^} 
r  -  1  t 


For  fixed  t  the  variables  W^*^  are  independent  Gaussian  variables 

with  mean  0  and  variance  t(l-t).  It  follows  that 

M  =  -  M  f  J -  t  -  dt  =  -  ?  M  as  claimed, 
r  r  JQ  1  -  t  2  r 

We  now  show  that  the  limit  step  in  (2.12)  is  valid.  We  will 

r  i  —  D  — O 

argue  in  the  function  space  D[0,1J.  Note  first  that 

implies  min  X  (i)  -*■  min  W  (i)  in  D[0,l].  Next  consider  the  contin- 
-  n  t  ..  t  ,i_c 

11  i  **  f(tl 

uous  functional  Tg  :  D[0,l]  R  defined  by  Tg(f)  =  |  y— -  dt. 

Since  min  X  (i)  is  piecewise  constant  and  equals  min  X.(i)  on  the 
.  n  t  •  1 

1  i ... i  + 1 


interval  <  t  <  J-~  %  we  have  that 
n  —  n 


T  (min  X(i» 
t  j  n  t 


€n<j<(l-€)n 


min(nXj/n(i)) 


i+1 

n 


i 

n 


1-  t 


dt 


€n<j<(l-€)n 


min(X^(i))(-  log  (1  -  - r) ) 


min  X . (i)  min  X.(i) 

E  - J--Y-  +  0  E - 

€n<j<(l-€)n  n  3  +  1  6\  (n-j)2 
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E(M^)  ri— fc  timin  v 
n-i  +  1  ^  J  1- t 


(2.17) 


1-e  E(min  W  ) 


€n<i<(l-€)n 


dt 


To  prove  (2.12)  note  that 


fl  E(min  Kt(l)) 


!0 


1  -  t 


dt 


is  a  convergent  integral  so  the  right  side  of  (2.17)  approximates 
this  arbitrarily  well  for  €  sufficiently  small.  Further 


E<>!  )  |i  l  j-j- 

Z  - t~T  <  £  - ^TT  <  c  Z  -  / — ~r 

n  -  i  +  1  •  —  .  n-x  +  1-  .  n/n-x 


E(iM.|) 


i<6n 


i<€n 


i<€n 


for  some  positive  c.  The  last  sum  is  a  Reimann  sum  for 


r£  / - 

/  x 

inl¬ 


and  so  can  be  made  arbitrarily  small  for  small  €.  The  same  argument 
works  for 


E(M.) 

I  - 7TT  • 

ri  ^ ■  n  -  i +  1 

(l-6)n<i 


This  completes  the  proof  of  (2.12)  and  thus  of  Theorem  3. 
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3.  Yes-No  Feedback 


In  this  section  we  discuss  problems  concerning,  a  deck  of 
n  cards  with  c^  cards  of  type  i,  1  <  i  <  r.  We  again  write  ~  for  the 
composition  vector  c  =  (c^,...,cr).  On  each  trial  the  subject  is 
told  if  the  previous  guess  was  correct  or  not.  We  refer  to  this 
situation  as  yes-no  feedback.  The  problem  is  complicated  when 
max(c)  >  1,  so  we  first  state  results  for  a  deck  of  n  cards  labeled 
l,2,...,n.  We  begin  with  no  feedback  and  complete  feedback  guessing 
and  compare  these  to  yes -no  feedback. 

No  feedback.  If  no  feedback  is  provided,  then  any  guessing 
strategy  has  one  correct  guess  as  its  expected  value.  Several 
writers  have  shown  that  the  variance  of  the  number  of  correct  guesses 
is  largest  when  the  guessing  sequence  is  a  permutation  of  {l,2,...,n} 
(see  J.  A.  Greenwood  (1938)  and  the  references  cited  there). 

Complete  feedback.  If  the  subject  is  shown  the  card  just 
guessed  each  time,  then  the  optimal  strategy  is  to  guess  a  card  known 
to  remain  in  the  deck.  The  number  of  correct  guesses  has  the  same 
distribution  as  a  sum  of  n  independent  random  variables  X^,l  £  i  <  n 
where  p(X^  =  i)  =  j  =  1  -  p(X^  =  0).  For  large  n  the  number  of  correct 
guesses  is  approximately  normally  distributed  with  mean  log  n  and 
standard  deviation  Aog  n. 

If  the  subject  is  only  given  yes-no  feedback,  then  the 
optimal  and  worst  case  strategies  are  described  by  the  following 
pair  of  theorems. 
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Theorem  5.  For  a  deck  containing  n  cards  labeled  (l,2,...,n)  a 
guessing  strategy  which  maximizes  the  expected  number  of  correct 
guesses  when  yes-no  feedback  is  available  is  the  strategy  which 
guesses  type  1  until  the  guess  is  correct,  then  guesses  type  2  until 
the  guess  is  correct  (or  the  end  of  the  deck  is  reached)  and  so  on. 
If  G  denotes  the  number  of  correct  guesses  under  this  strategy,  then 


(3.1) 


P(G>  k)  =  -~r  k  =  1,2 . n 


(3.2)  E(G)  =  l+  -r+...+A-=e-l  +  0(A) 

Z .  n .  n . 


Theorem  6.  For  a  deck  containing  n  cards  labeled  (l,2,...,n)  a 
guessing  strategy  which  minimizes  the  expected  number  of  correct 
guesses  when  yes-no  feedback  is  available  is  the  strategy  which 
guesses  type  i  on  the  iC^  trial  until  a  guess  is  correct  and  then 
repeats  the  correct  guess  for  the  remaining  trials.  If  g  denotes  the 
number  of  correct  guesses  under  this  strategy,  then  g  takes  values 
zero  and  one  with  probabiltiy: 


(3.3) 


P(g  =  0)  »  A  +  0(A)  -  1  -  P(g  -  1) 
e  n . 


(3.4) 


E(g)  =  1  -  -J  +  0(— j-)  . 


Theorems  5  and  6  deal  with  the  only  type  of  deck  where  we  can  pro¬ 
vide  a  simple  description  of  the  optimal  strategy.  In  each  case  the 
optimal  stratev  is  the  "greedy"  strategy  which  guesses  the  most 
probable  (for  Theorem  6  the  least  probable)  type.  U’e  do  not  know  if  the 
greedy  strategy  is  optimal  for  decks  of  2n  cards  with  composition 


vector  (2,2,..., 2).  We  will  show  that  the  g?eedy  strategy  is  not 
optimal  for  the  9-card  deck  with  composition  vector  (3,3,3).  We  first 
need  some  notation. 

Let  p  =  (p^,p2,...,pr)  be  a  vector  with  integer  components 
Pj  h  0*  Define 

(3.5)  N(c;p)  =  the  number  of  permutations  of  c^  +  ...  +  symbols 
which  do  not  have  symbol  1  in  the  first  p^  positions,  nor  symbol  2 
in  positions  p^  +  l.-.-.p^,  etc. 

Thus,  N(c;0)  =  (c^  +  ...  +  c^.)!.  The  numbers  N(c;p)  allow 
computation  of  the  most  probable  type  at  any  stage  of  an  experiment 
with  yes-no  feedback.  They  are  closely  related  to  rook  polynomials 
described  in  Chapters  7  and  8  of  Riordan  (1958)  and  are  discussed 
further  in  Diaconis,  Graham,  and  Mallows  (1979). 


Algorithm  to  compute  probabilities  with  yes-no  feedback.  Suppose  an 

experiment  started  with  composition  vector  c^  and  that  after  the  j*"*1 

guess  there  have  been  Y^(i)  yes  answers  on  type  i  and  p^(i)  no 

answers  on  type  i,  1  i  <_  r.  The  deck  now  has  composition  vector 

c  =  cq  ~  Yj.  We  will  call  c  the  reduced  composition  vector. 

Writing  for  the  vector  (0  ...  1  ...  0)  with  a  1  at  position  i  and  0 

elsewhere,  the  conditional  probabilities  of  a  correct  (or  incorrect) 

guess  on  type  i  on  the  j  +  1st  trial  given  Y  and  p.  are: 

3  3 


(3.7) 


_  N(c;p  +6  ) 
P(no  on  type  i|Y.;p.)  =  - - 


3  3 


N(c;p..) 


For  1  £  i  <  r. 

As  implied  by  (3.6)  and  (3.7),  the  function  N  satisfies  the 
recursion 

(3.8)  N(c;p  +  <$,  )  =  N(c;p)  -  c  N(c  +  S,  ; p) ,  1  <  k  <  r 


with  N(c;0)  =  (c^  +  . . .  +  c  ) I 


This  recursion  can  be  solved  in  closed  form  to  allow  computation  of  N 


(3.9)  N(c;p)  =  E  (-1) 

j.<c 


il+  •**  +  ir/'P 


rP-\  ((c-,-1,)  +  ...  +  (c^-ir)) 

7T 


(P1\  (PA  uc 

VV* "W  (cl“  V1  (Cr'i, 


The  proof  of  (3.9)  is  given  in  Diaconis,  Graham,  and  Mallows  along 
with  a  host  of  other  properties  of  N(c;p). 

Let  E(c;p)  be  the  expected  number  of  correct  guesses  under  an 
optimal  strategy  starting  from  the  reduced  composition  vector  c. 
E(c;p)  is  well  defined  since  there  are  only  a  finite  number  of  stra¬ 
tegies  and  one  (or  more)  of  them  maximizes  the  expected  number  of 
correct  guesses.  It  is  straightforward  to  show  that  E  satisfies  the 
recurrence: 

(3.10)  E(c;p)N(c;p)  =  max(E(c;p + 5^)N(c;p + 5^) 

k 

+  E(c  -  6,  ;p)c,  N(c  -  5,  ;p)  +  c,  N(c-6.  ;  p) ) 


where  N(c;p)  was  defined  in  (3.5).  We  have  not  been  able  to  solve 


this  recurrence  for  E(c;0)  in  closed  form  even  though  N  is  known 
through  (3.9).  The  recurrence  can  be  solved  numerically.  For 
example,  Mary  Ann  Gatto  (Gatto  (1978))  generated  values  for  all 
composition  vectors  smaller  than  (5, 5, 5, 5, 5).  Some  results  are: 

E(3, 3, 3;0)  =  4.26,  E(4,4,4,4;0)=  5.47,  E(5,5,5,5,5;0)  =  6.63  . 

The  details  of  computing  a  number  like  E(5,5,5,5,5;0)  are  not  simple. 
The  computation  required  15  hours  of  cpu  time  on  a  Honeywell  6070 
computer  along  with  clever  use  of  both  recursions  (3.8)  and  (3.10). 

The  optimal  strategy  at  each  stage  is  determined  by  finding  a 
k  which  maximizes  the  right  side  of  (3.10).  Formula  (3.6)  implies 
that  the  greedy  strategy  at  each  stage  is  determined  by  choosing  a 
k  maximizing  c^  N(c-6^;p).  We  now  give  an  example  which  shows  that 
the  greedy  strategy  is  not  optimal. 

Consider  a  9-card  deck  with  3  each  of  3  different  types  of 
card.  A  complete  listing  of  N(c;p)  and  E(c;p)  for  all  (c;p)  that 
arise  with  this  9-card  deck  is  given  in  Diaconis  and  Graham  (1978). 

In  the  situation  summarized  by  (231;003)  the  optimal  strategy  is  to 
choose  type  3  on  the  next  guess.  However,  type  2  is  more  probable 
than  type  3  on  the  next  guess.  The  situation  summarized  by  (231;003) 
could  arise  under  the  optimal  strategy  from  starting  position 
(333;000)  as  follows:  the  first  guess  is  type  1,  and  this  is  correct. 
The  next  three  guesses  are  type  3,  and  all  three  guesses  are  wrong. 

The  next  guess  on  type  3  is  correct.  At  this  point  the  situation  is 
summarized  by  (232;003)  and  the  optimal  guess  is  type  3.  If  this  is 
correct,  then  the  situation  is  summarized  by  (231;003). 
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Even  though  the  greedy  strategy  is  not  optimal,  computations 
reported  in  Diaconis,  Gatto,  and  Graham  (1979)  show  that  the  expected 
number  of  correct  guesses  under  the  greedy  strategy  is  extremely 
close  to  the  expected  number  under  the  optimal  strategy  for  decks 
with  composition  vector  (3,3,3)  or  (5, 5, 5, 5, 5). 

If  e(c;p)  is  the  expected  number  of  correct  guesses  for  the 
worst  possible  strategy,  then  e(c;p)  satisfies  a  recurrence  obtained 
from  replacing  max  by  min  in  (3.10).  We  have  not  pursued  the  problem 
of  numerical  computation  of  e. 

Even  though  the  optimal  strategy  seems  to  be  extremely 
complex,  we  believe  that  the  following  simple  persistence  conjecture 
holds:  In  any  problem  with  partial  feedback,  if  symbol  1  is  the 
optimal  guess  on  trial  i  and  a  guess  of  1  is  answered  by  "no,"  then 
symbol  1  is  optimal  on  guess  i  +  1. 


Proofs  for  Section  3. 

Proof  of  Theorem  5.  When  the  given  strategy  is  used,  the  permutations 

with  k  or  more  correct  guesses  are  those  in  the  set 

=  {it  :  rr  1  ( 1 )  <  7T~ 1  (2)  <...<  7T  1  (k) }  .  Thus,  P(G>k}  =  POireA^  =  ^  . 

This  proves  (3.1)  and  implies  (3.2). 

We  now  argue  that  the  outlined  strategy  is  optimal.  In  this 

problem  a  strategy  S  may  be  regarded  as  a  sequence  of  n  functions 

S  =  (S,  ,S„, . . .  ,S  )  where  S.  :  {0,l}i  *  {l,2,...,n}.  The  interpreta- 

l  l  n  l 

tion  is  that  a  point  in  (0 , l) 1  1  represents  a  sequence  of  i-1  yes  or 
no  answers,  0  standing  for  no  and  1  for  yes.  The  expected  value  of  a 
strategy  is  E(S)  =  E^ir(i)S  ^  where  is  one  or  zero  as  i  =  j 
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or  not.  We  will  say  that  strategy  S  dominates  strategy  S'  if 
E(S)  E(S').  Strategies  S  and  S'  will  be  called  equivalent  if 
E(S)  =  E(S'). 

We  first  argue  that  the  given  strategy  calls  the  most 
probable  symbol  at  each  stage.  This  is  implied  by  the  following 
monotonicity  property  of  the  function  N: 

(3.11)  an<*  only  if  N(l;p  +  6j)  <  N(l;p  +  Sj) 

This  property  of  N  is  proved  and  further  discussed  in  Diaconis, 

Graham,  and  Mallows  (1979).  Inequality  (3.11)  implies,  and  is 
implied  by,  the  following  combinatorial  fact  which  was  first 
established  by  Efron  (1963). 

(3.12)  (Efron's  Lemma).  Let  two  decks  of  n  cards  be  prepared.  The 
first  deck  labeled  (l,2,...,n),  the  second  deck  labeled  (a^.a^, . . . ,an) 
with  a^ e  (l,2, . . . ,n) .  Each  aeck  is  mixed  and  the  cards  turned  over 
simultaneously,  one  pair  at  a  time.  The  probability  of  no  matches  is 
largest  if  and  only  if  there  are  no  repeated  symbols  among  the  a^. 

That  is,  if  {a^}  =  (l,2,...,n). 

We  have  thus  argued  that  the  given  strategy  calls  a  most 
probable  symbol  at  each  stage.  We  want  to  show  that  any  strategy 
which  achieves  the  maximum  number  of  correct  guesses  in  this  problem 
has  this  property.  We  note  that  a  maximizing  strategy  exists  since 
there  are  only  finitely  many  strategies. 

To  begin  with  we  may  restrict  attention  to  strategies  which 
do  not  guess  symbols  known  not  to  be  left  in  the  deck  since  such 
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strategies  may  be  improved  uniformly  over  all  permutations  by  modi¬ 
fying  them  to  guess  only  symbols  which  have  not  been  definitely 
eliminated . 

We  will  argue  by  backward  induction  that  any  strategy  can  be 
strictly  improved  by  being  modified  to  choose  a  most  probable  symbol 
at  each  stage.  This  is  clear  at  trial  n  since  modifying  a  strategy  S 
so  that  it  chooses  the  most  probable  symbol  on  the  final  guess  can 
only  increase  E(S) .  Consider  a  strategy  S  which  chooses  the  most 
probable  symbol  on  trials  n-k,  n  -  k  4- 1, . . .  ,n,  for  fixed  k  >  0. 
Consider  a  history  h  €  (0,l}n  ^  ^  for  which  Sn_^_^(h)  =  a  where  b  ^  a 
is  the  most  probable  guess  and  strictly  more  probable  than  a.  By 
(3.11)  we  must  have  p^  >  pa,  i.e.,  p^  >  pQ  +  1.  No  matter  what  the 
outcome  of  the  guess  S  k_^(h)  =  a  is,  no  symbol  is  more  probable 
than  b  just  before  trial  n-k.  Thus,  by  induction  we  may  assume 
S(h,0)  =  S(h,l)  =  b  (i.e.,  we  can  modify  S  to  have  this  prop..  \y 
without  decreasing  E(S)). 

Consider  the  portion  of  the  "strategy  tree"  of  S  following  h 

A  A. 

(see  Figure  1).  Form  the  strategy  S  from  S  by  defining  S  ,  (h)  =b, 

n  k*- jl 

A  A 

Sn_^(h,0)  =  S^_k(h,l)  =  a  and  interchanging  the  two  parts  Tq^  and  T^q 
of  S  which  follow  (h,0,l)  and  (h,l,0)  (see  Figure  1). 

We  claim  that  for  each  permutation  ir  of  the  deck  there  is  a 

A 

unique  permutation  it  of  the  deck  such  that  the  number  of  hits  that  S 

A  A 

has  for  u  is  the  same  as  the  number  of  hits  that  S  has  for  ir.  This 
correspondence  is  given  by  switching  coordinates  n-k-1  and  n-k: 


Figure  1 


i.e. , 


ir(n  -  k)  for  i  =  n  -  k  -  1  , 

A 

ir(i)  =  <  Tr(n-k-l)  for  i  =  n - k  , 


{  ir(i) 


otherwise 


1=  a 


It  is  now  a  simple  matter  of  checking  the  four  cases  iT(n-k-l)< 

/-  b  A  W  a 

iT(n  -  k)\  to  see  that  S  has  the  desired  property  or  tt.  For 

W  b 

example,  if  ir(n-k-l)  =  a,  7r(n-k)  ^  b  (and,  of  course,  it  generates 


the  history  h) ,  then  tr  generates  the  history  (h,l,0),  collects  one 
more  hit  (at  the  question  Sn_jl_^(h)  =  a)  and  exits  into  T^q. 


However,  in  S,  it  gets  a  no  at  the  question  “  b*  a  yes  at 


the  question  S  .  (h,0)  =  a  (collecting  one  hit)  and  also  exits  into 


E(S)  >  E(S) 


However,  by  induction  if  we  replace  ^(h,0)  =  a  by  -  b, 

then  since  b  is  (still)  more  probable  than  a,  this  gives  a  strict 

A 

improvement  to  S.  This  shows  that  an  optimal  strategy  must  also 
guess  the  most  probable  symbol  on  trial  n-k-1.  This  completes  the 
induction  step  and  the  theorem  is  proved. 


Proof  of  Theorem  6.  Under  the  given  strategy  the  number  g  of 
correct  guesses  is  either  zero  or  one.  The  probability  of  one  cor¬ 
rect  guess  is  the  probability  that  two  permutations  have  one  or  more 
matching  coordinates.  This  probability  is  well  known  (Feller  1968, 
p.  100)  to  be 

P(g  =  l)  =  1-P(g  =  0)  =  1- A  +  ~+  ...  +  (-l)"'1  A=l-~  +  0(A)  . 

j.  n.  e  n. 

This  proves  (3.3)  and  (3.4). 

We  now  show  that  the  strategy  given  in  Theorem  6  achieves 
the  minimum  number  of  expected  correct  guesses. 

Using  the  notation  established  in  the  proof  of  Theorem  5, 
a  strategy  S  is  a  sequence  of  functions  S  =  (S^.S^, . . . ,Sn) , 

:  {0,l}*  *  -*■  {l,2,...,n).  To  begin  with,  it  is  easily  shown  that 
the  expected  value  of  any  strategy  can  be  decreased  by  modifying  it 
so  that 


(3.13)  Sj.(0 . 01)  =  Si_1(0,...,00)  for  i  =  2,3,...,n 


and  so  that  S  never  achieves  more  than  1  correct  guess. 
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Restricting  attention  to  strategies  which  satisfy  (3.13)  we 

see  that  the  strategy  S  is  determined  by  the  n  numbers 

S^,S2(0) ,S^(0) , . . . ,Sn(0) .  The  expected  value  of  S  is  the  probability 

of  one  or  more  matches  of  a  random  permutation  f  to  n  symbols  labeled 

S^^CO) , . . .  »Sn(0) .  Efron's  Lemma  (3.12)  shows  that  this  probability 

is  smallest  when  (S- ,S„(0) , . . . ,S  (0)}  =  {l,2,...,n}.  This  proves 
x  2.  n 

Theorem  6. 
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4.  Evaluation  of  Feedback  Experiments 

The  evaluation  of  feedback  experiments  is  problematic  because 
it  is  impossible  to  know  what  use  a  subject  will  make  of  the  feedback 
information.  In  this  section  we  introduce  an  evaluation  approach 
called  skill  scoring.  The  idea  is  to  compare  the  number  of  correct 
guesses  with  a  base  line  rate  calculated  from  the  conditional  expected 
number  of  correct  guesses  given  the  available  information. 

One  example  of  skill  scoring  in  the  present  setting  was  given 
in  Table  1.  To  motivate  the  abstract  definitions  we  are  about  to 
present,  we  review  this  example.  The  problem  considered  was  card 
guessing  with  two  types  (call  them  type  1  and  type  2) ,  k  of  each  type 
(so  n=2k  cards  in  all)  and  complete  feedback.  We  can  model  this  by 
considering  the  basic  probability  space  to  be  $n,  the  set  of  permuta¬ 
tions  on  {l,2,...,n},  with  the  uniform  probability  measure.  A  permu¬ 
tation  ir  is  chosen  at  random  and  the  ith  trial  is  declared  "type  1" 
if  tt (i)  is  odd  and  "type  2"  if  n(i)  is  even.  On  the  ith  trial  the 
guessing  subject  is  given  feedback. 


f. 

l 


ith  guess  is  correct, 
ith  guess  is  incorrect. 


This  particular  feedback  function  only  depends  on  the  current 
coordinate.  Some  possible  variations  are: 


(4.1a)  Feedback  might  depend  on  previous  outcomes.  This  is  realis¬ 
tic  in  card  guessing  experiments  with  unconscious  cuing  due  to  sub¬ 
jects  being  within  sight  or  earshot  of  one  another.  If  there  were 
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few  correct  guesses  in  the  early  stages,  more  active  feedback  might 
be  made  available  as  the  experiment  progressed. 


(4.1b)  In  addition  to  telling  if  the  previous  guess  was  correct  or 
not,  feedback  might  indicate  if  incorrect  guesses  were  "close.” 

Fisher  (1924),  (1928),  and  (1929)  gives  some  examples  of  measures  of 
closeness. 

(4.1c)  Feedback  might  only  be  available  on  some  outcomes.  For 

instance,  the  subject  might  be  given  feedback  after  red  guesses  but 

no  feedback  after  black  guesses. 

We  formulate  the  general  situation  in  terms  of  S  ,  the  set  of 

n 

permutations  of  {l,2,...,n}  =  To  model  a  pack  of  cards  with  c. 

cards  labeled  i  we  need  the  idea  of  an  evaluation  function. 

For  example,  to  model  red-black  card  guessing  we  might  consider 

/ 

j  1  if  ir(i)  is  odd 

A  ("(!))  =  ( 

1  2  if  if(i)  is  even 

(4.2a)  An  evaluation  function  A  is  a  sequence  of  functions 

A  =  (Aj,...,An)  where  A^(tt)  =  X^(Tr(i))  for  tt  e  S^.  Let  the  range  of 

A.  be  denoted  by  R  =  (A.  (ir(i))  :  n  e  S  }.  An  evaluation  function  is 
_ ±  ii  n 

of  fcype  r  if  A1(n)  -  1  =  Tr(i)(mod  r).  Let  A  denote  the  algebra  in 
Sn  generated  by  Aj.Aj,  . . .  ,A. . 

We  will  restrict  attention  to  guessing  strategies  which  take 
values  in  R^.  For  each  sequence  of  guesses  and  each  history  up  to 
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time  i,  we  must  define  a  feedback  function  f^.  For  complete  feedback 

guessing,  f.  =  X.(tt).  For  yes-no  feedback  f.  =  6,  ,  where  G.  is 

i  i  iA*b.  i 

,  , th  1  1 

the  i  guess . 

(4.2b)  A  feedback  function  f  is  a  sequence  of  functions 

f  =  (f . .  ,f  )  where  f .  :  R,  xr  xr  xs  ->-0.  For  each  fixed 

1  n  i  1  2  i  n  n 

r  =  (ri»r2»  •  •  •  .r^),  we  may  regard  f.^  as  a  function  f^ (r^, . . .  ,r^;  •) 

from  S  into  0,  .  This  function  is  to  be  measurable  when  S  is 
n  n  n 

equipped  with  the  algebra  defined  in  (4.2a)  for  any  r.  We  also 
define  the  algebra  3(r)  =  o{f  ^(r^^;  • )  ,f2^ri’r2’ '^  ’  *  *  *  ’ 

f .  (tj, . . • j  * ) } • 

This  frightening  terminology  has  the  following  interpretation: 

that  f  is  measurable  means  that  ^  only  depends  on  the  first  i  guesses 

and  the  values  A.  (ir)  , . . .  ,A .  (tt)  .  A  function  from  S  will  be  measura- 
-*-1  n 

ble  with  respect  to  3(r^,...,r^)  if  it  only  depends  on  the  first  i 
components  of  the  permutation  through  the  feedback  information  given 
when  guesses  ri»r2’***»ri  are  made  on  trials  l,2,...ti. 

(4.2c)  A  feedback  function  is  adapted  if  S_  ,  is  3(r . r.) 

Vi  1  1 

measurable  for  each  r^,^, . . . ,r^,  1  £  i  <  n.  Adaptability  means 
that  the  feedback  includes  the  information  that  the  last  guess  was 
correct  or  not. 

(4. 2d)  A  guessing  strategy  g  is  a  sequence  of  functions 
8  =  (g1»g2»---»8n)  where  gj  is  a  constant  and 

g.  :  R,  *  ...  *  R.  ,  XS  -*■  R.  satisfies  g.(r, ,...,r.  *)  is 

i  1  i—JL  n  1  li  i*i 

3(rp . . .  ,r^_^)  measurable.  The  value  of  g^  will  be  denoted  G^. 
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(4.2e)  The  collection  of  functions  X,f  will  be  called  an  experiment. 
We  define  the  skill  scoring  statistic  for  an  experiment  by 


(4.3)  S  =  I  (SG  A  -E{6g  x  |3(G1,...,Gi_1)}} 

i_l  ^  uiAi 


The  main  motivation  for  considering  S  is  that  for  a  wide 
variety  of  experiments  S  can  be  normed  to  have  an  approximate  standard 
normal  distribution  uniformly  in  guessing  strategies.  This  is  made 
precise  in: 


Theorem  7.  For  an  experiment  as  defined  in  (4.2e)  and  any 
guessing  strategy  g ,  the  skill  scoring  statistic  S  defined  by  (4.3) 
satisfies 


(4.4) 


E(S)  =  0  . 


If  the  evaluation  function  is  of  type  r  as  defined  by 
(4.2a)  and  the  feedback  function  adapted  as  defined  by  (4.2c), 
then  as  n  tends  to  infinity. 


(4.5) 


/ni(l-i)  ) 


dt  . 


Convergence  in  (4.5)  is  uniform  in  guessing  strategies  g. 

We  now  discuss  some  motivation  and  properties  of  S.  In  the 

absence  of  "talent,”  the  distribution  of  5  .  given  the  feedback 

GiAi 

information  is  the  conditional  permutation  distribution.  S  will  be 


large  when  there  are  more  successful  guesses  than  chance  predicts. 


To  compute  S,  only  the  observed  guesses  need  be  known, 

not  the  entire  guessing  strategy. 

For  definiteness  consider  the  example  in  Table  1— card 
guessing  with  complete  feedback  from  a  deck  containing  k  red  and 
k  black  cards.  As  shown  in  Theorem  1,  a  subject  using  the  optimal 
(or  worst  case)  strategy  expects  to  obtain  approximately  k  +  —  »^rk 
(or  k  -  /irk)  correct  guesses.  The  statistic  S  compensates  for  this 
by  subtracting  a  random  correction  factor  with  mean  value  k  +  /irk 
(or  k  -  /itk) .  This  allows  us  to  see  if  the  subject  scored  more 
than  chance  when  the  strategy  has  been  adjusted  for.  The  conditional 
expected  value  in  (4.3)  may  be  complicated  to  compute  if  f  is 
complex.  For  yes-no  partial  feedback  the  conditional  expectations 
may  be  computed  using  (3.6)  and  (3.7). 

One  penalty  that  must  be  paid  for  the  close  tracking  of  6  1 

Vi 

by  its  expected  value  is  as  follows:  If  the  feedback  information  at 
some  stage  determines  the  composition  of  the  remainder  of  the  deck, 
none  of  the  subjects'  guesses  from  that  trial  on  have  an  effect  on  S. 
This  can  be  seen  in  the  last  guess  in  Table  1  when  the  feedback 
information  determined  that  the  last  remaining  card  was  black. 
Similarly,  the  possible  corrections  due  to  feedback  are  less  pro¬ 
nounced  at  the  beginning  of  the  deck  and  more  pronounced  toward  the 
end  of  the  deck. 

Theorem  7  holds  because  the  terms  in  the  sum  for  S  are  a 
Martingale  difference  sequence  with  well-behaved  variance.  The 
Martingale  central  limit  theorem  is  in  force.  If  there  was  a  prac¬ 
tical  reason  for  doing  so,  the  result  could  be  extended  to  scoring 
functions  of  the  form 
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(4.6)  s  -  £  {W  (G  ,...,G  ;  A  ,...,A  ) 

i=l  1  1 

—  EfW^(G^,. . .,G^;  A^, ... ,Aj) |3(G^,...,G._j)l} 

where  the  functions  W^  could  be  chosen  to  give  desired  weights  to 
correct  or  incorrect  guesses  depending  on  previous  results. 

We  note  that  the  form  and  motivation  for  the  statistic  S  are  quite 
similar  to  the  form  and  motivation  for  the  Kantel-Haenszel  statistic 
as  discussed  (for  example)  by  Tarone  and  Ware  (1977).  It  should  be 
possible  to  show  that  S  is  locally  most  powerful  by  arguments 
similar  to  those  used  to  show  that  the  Mantel-Haenszel  statistic  is 
locally  most  powerful  against  Lehmann  alternatives. 

We  now  illustrate  the  hypothesis  of  Theorem  7  through  some 

examples .  * 

! 

» 

(4.7)  Example  of  the  need  of  adaptability  assumptions. 

I 

The  adaptability  assumption  (4.2c)  simply  means  that  the 

I 

feedback  includes  the  information  that  the  last  guess  was  correct  or  | 

? 

i 

not.  To  see  that  there  is  no  hope  of  a  normal  limiting  result  with-  I 

out  this  assumption,  consider  an  experiment  with  no  feedback 

information,  for  example,  f.  =  1.  To  be  specific,  suppose  there  are 

n  each  of  two  types,  and  that  the  guessing  strategy  always  guesses 

type  1.  Then  the  number  of  correct  guesses  will  always  be  n,  and 

the  conditional  probability  subtracted  off  at  each  stage  will  always 

equal  1/2  so  that  S  =  0.  This  example  presents  a  fundamental  problem 

for  the  widely  used  normal  approximation  to  classic?!  card  guessing 
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experiments  without  feedback  (this  is  discussed  by  Greville  (1941), 
(1944)).  It  underscores  the  need  for  common  sense  even  when 
Theorem  7  is  in  force  since,  if  a  subject  always  guesses  the  same 
type  of  card,  the  randomness  captured  by  the  limiting  normality  will 
be  due  to  the  fluctuation  of  the  conditional  expectations  in  S. 

The  next  example  shows  the  need  for  the  assumption  of  a  deck 
of  type  r  by  exhibiting  several  non-normal  limits  (depending  on  the 
guessing  strategy)  for  a  deck  labeled  {l,2,...,n). 


(4.8)  Example;  Partial  Feedback  guessing  for  a  deck  labeled 
(l, 2,3, . . . ,n} . 


In  this  problem,  as  discussed  in  Section  3,  a  deck  of  n  cards 
is  labeled  (l,2,...,n).  A  subject  guesses  the  value  of  each  card 
sequentially  and  is  told  if  each  guess  is  correct  or  not.  Here 


A±  (IT  (i) )  =  TUi),  f1(G1,...,Gi;  IT  (1) , . . .  ,'if  (i) )  =  6^^  ,  and  S  can  be 


represented  as 


n 

Z  {6 


Z  <5, 


^  ^uU)  n-i+1  ^  uGilT(j) 


To  see  that  the  distribution  of  S  depends  on  the  guessing  strategy  we 


consider  three  cases: 


Case  I.  Worst  case  guessing.  If  the  guessing  strategy  is  the  worst 

case  strategy  established  in  Theorem  6,  we  will  show  that  the 

limiting  distribution  of  S  converges  to  a  beta  distribution  on 

(1+-^)  to  1  with  an  atom  at  -  (1--^-).  More  precisely, 

e  e 

(4.9)  P(S£t)  -*•  G(t)  as  n  tends  to  infinity  where  the  distribution 

function  G(t)  is  defined  by 


G(t)  =  0  for  t  <  (1  -  , 

e 


-  for 
e 


-  i  (1  -  -4)  <  t  <  £  <1  +  -4)  , 


n  (t  -  \) 


1/2 


for  |  (l  +  -7)  £  t  <  1  , 
e 


=  1  for  t  >  1 

Case  II.  G^  =  1.  We  will  show  that  when  G^  always  guesses  1,  the 
distribution  of  S  converges  to  an  exponential  distribution  on  (-°°,l]< 
More  precisely. 


(4.10) 


P{l-S<x}  *  1  -  e  x  for  0  <  x  <  00  . 


Note  that  while  the  expected  value  of  S  agrees  with  the  limiting 
expected  value  of  0,  computation  shows  that 

Var(S)  =  2  log  n  +  ,  as  n  tends  to  infinity. 


Case  III.  Best  case  guessing.  In  Theorem  5  the  rule  for 


maximizing  the  expected  number  of  hits  was  shown  to  be  the  rule  which 
guesses  the  most  probable  card  at  each  stage.  When  this  rule  is 
used,  we  will  show  that,  as  n-*00,  the  statistic  S  tends  to  a  countable 
mixture  of  continuous  distributions: 


(4.11)  P(S<t)  =  Z  p  F  (t) 

i=l  1  1 


where 


P 


i 


1  1 
i!  '  ( i— 1)  ! 


i+1 

f  (t)  =  p{  n  l 
1  j-1 


t-i-, 

i  <  e  > 


where  . . .  ,Li+^  are  the  lengths  of  the  i  +  1  intervals  the  unit 

interval  is  partitioned  into  by  dropping  i  points  at  random. 


Proofs  for  Section  4. 


Proof  of  Theorem  7.  Consider  the  basic  probability  space  with  the 


uniform  distribution.  Let  G- , ,...,G  be  any  sequence  of  guesses. 

■*•11  r 


Let  Bq  =  (4>,Sn},  B±  =  3=(G1,...,Gi+1)  for  i  =  1,2, . . .  ,n-l,  Bn  =  2  n. 


Thus ,  B.cb,  c,..cb  .  Let 
u  i  n 


zi  *  ~pi — f-( v,  ■  8{V ,|Bi-i}>  and  xi  ■  A  zj  • 

A  HI--)  11  i~1 


r  r 


Because  f  is  adapted,  X^  is  a  Martingale  with  E(X^)  =  0.  To  prove 


(4.5),  we  first  show  that  (4.5)  holds  when  f ^ and  G^  is  the  result 


of  best  case  guessing.  Further >  and  without  real  loss,  suppose  that 
n  =  rk.  Let  denote  the  minimum  of  the  number  of  each  type  seen 
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The  inequality  (2.13)  implies  that  there  is  a  positive  constant  such  that 


E(|Mi|)  <  c  J  —  ~n  —  ■  .  Using  this  and  Markov's  inequality  it 
follows  that  for  any  6  >  0, 


In  ^  n  -  i  +  1 


>  €>  < 


€  v^n 


so  that  (4.12a)  is  true.  The  proof  of  (4.12b)  is  similar.  Hence,  we 
have  shown  that  (4.7)  holds  when  f^  =  X^  and  is  best  case  guessing. 
A  similar  proof  works  if  f^  =  X^  and  is  worst  case  guessing.  If 
now  f  is  an  arbitrary  measurable  feedback  sequence  and  G^  an  arbi¬ 
trary  guessing  strategy,  let  p..  =  E{<$g  ^  | B±> .  Recall  i  defined  in 
(4.2a).  Let  Pi  =  E(6^  x JA^},  p1  =  E(6G  |A  ).  Then  ^  <  p  <  p  ; 

and  since  (4.12a)  and  (4.12b)  hold  for  p^  and  p^,  they  must  hold  for 
p^.  This  completes  the  proof  of  Theorem  7. 

Proofs  for  example  (4.8). 

Proof  of  (4.11).  For  worst  case  guessing  S  takes  values  which  depend 
on  T,  the  time  of  the  first  correct  guess.  Let  N(i,n)  denote  the 
number  of  permutations  it  £  which  do  not  have  ir(j)  =  j,l  <  j  _<  i. 
Equation  (3.9)  implies  that  N(i,n)  =  E*_q(-1)^ (*) (n  -  j) !  and  we 

see  that  P{T  =  k)  =  ^  N(k-1,  n  -  1)  and  PCi^  guess  is  correct  | past)  = 

N(i  -  1,  n-  1) 

N(i  -  1,  n) 


n: 


Thus,  S  takes  values 


o 


(l-~-— -^o) 

n  /  i  \  t 


(n-l) 


with  probability  —  , 


n  —  9 

with  probability 
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-j 


£ 

1  -  E  with  probability  ^  N(i-1,  n-1)  , 

iui-1,  n)  n. 


1  -  l  with  probability  A  N(n-1,  n-1)  , 

Nil— l,  nj  n. 


-  ^  -TO-ir^'  with  probabllti>'  ">  • 


1  11 

We  now  show  that  — r  N(i,n)  »  (1-*=*)  +  0(—)  uniformly  in  i« 

n.  n  n 


Indeed, 


*  - (i  s  1 


=  e  +  0(A)  -  (1-A  =  0(k  . 

n.  n  n 


Thus,  for  any  k,  1  <  k  <  n. 


y  N(i-1,  n-1)  _ 
^  N(i-1,  n) 


L  k  1 — 

I  l  - Ull 

n  i=i  i  -•! 

\  n 


1  11  2 

+  0(i)  =  i-i(l-~)  +  0( 

n  12  n 


so  that  S  takes  values 


k  ,  k 

■|  +  -|(1-^)  +  0(~)  with  probability  ^(1-^-)  +  0(-^) 

n 


for  1  <  k  <  n  and  S  takes  the  value  -  -|(1--^)  +  ^(~)  with  probability 

11  6 

~  +  0(A)  . 

e  n. 

Using  these  estimates  shows  that  P(S<t)  -*■  G(t)  for 
11  1 

t  -  2  +  2<1  — 2^  *  For  iar8er  t  we  have 
e 


sit- 


P(S  <  t)  -  i  +  PF  +  +  OF)  <  t)  +  0(-~r) 

-  e  lln  n  —  n. 


*  I  +  P{_  II  <  log(2(t  ~h  +  0  A}  +  0(-nr) 

e  n  —  2  t  n  n. 


=  i  +  (1  -  P{f  <  -  I  log  "(t  +  0t(i)»  +  0(i) 


=  ~  +  1  -  <1  +  0^)  +  ^  2(1 -i)  +  0(h 

e  e  n  n  n  n 

0<j<nf (i) 


1  1/2  i 

=  ^2(t-i)  +  OF) 

l  n 

where  we  have  written  f(t)  =  log  2(t--j). 
proof  of  (4.9). 


This  completes  the 


Proof  of  (4.10).  When  the  guessing  strategy  has  =  1,  then  S  takes 
values  1  -  (Hn~HT),  k =  0,1,2, ... ,n-l,  where  T  is  uniformly  dis¬ 
tributed  on  {0,1,2, ... ,n-l}  and  =  l+...+1/k.  So, 

HT“Hn  -t 

P{1-  S  >  t}  =  P{e  T  <  e  fc} 

H  — H 

=  P(e  T  n<e_t|v'n<T<n-  *^n)}  (1  +  0(— ))  +  0(— ) 

Jn  V^n 

log  T-log  n+0(i) 

=  P(e  <  e  |»^  <  T  <  n  -  vQ  +  0(— ) 

=  P{^(1  +  0(|))  <  e”'t  | l/n  <  T  <  n  -  /n)  +  0(— ) 
n  T  -  -  -  y£ 

e  C  as  n  tends  to  infinity. 


Proof  of  (4.11).  With  best  case  guessing  it  was  shown  in  Theorem  5 
that  the  number  of  correct  guesses,  G,  takes  value  i  with 
probability  p ±  =  jr  -  ,  1  <  i  <  n  - 1.  When  G  =  i,  let  be 

the  waiting  time  for  the  jth  correct  guess,  for  1  <  j  <  i.  The 


random  variables  ^(T^...,^,  n-^  +  .-.+T^  are  easily  shown  to 
have  as  limiting  distribution  the  distribution  of  the  lengths 
^1*^*2*  *  *  *  ’^i+1  *-he  i  i  intervals  that  the  unit  interval  is 
partitioned  into  by  i  random  points,  de  Finetti  (Feller  1971,  p.  42) 
has  shown  that  Ptl^  >  >  x£+1>  =  (1-x^  ... +xi+1)*  where 

+  denotes  positive  part.  When  G  =  i,  write  T  =  T±;  then 


i 

P{S<  t|G  =  i>  «  P{i  -  Z 

j=l 


T. 

J  1 
Z  - i - 

k=o  n-^‘k 


n-T 

E 

j=0 


n  -  i  -  k  — 


<  tj 


P{L1  l2 


T  ^  t-i-. 

Li+1  i e  1 


by  an  easy  argument.  This  completes  the  proof  of  (4.11). 
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